HttpClient 4.5.x教程

简介

之前一直使用httpclient进行爬虫的接口，但是一直对实现的底层原理不是很理解。尤其是httpclient这个对象十分庞大。
对这个类的详细的理解是出于对http底层的原理有了一定的理解之后，想着java这边对httpclient的封装又是如何支持http协议的，出于这个好奇心，就花了一天的时间去研究，总算有所收获。

如何构建一个可做请求的Httpclient

这节的内容主要参考了webmagic的源码里面的构造,主要涉及到三个主要对象的构建，HttpClient，HttpUriRequest，HttpClientContext。

HttpClient对象

private CloseableHttpClient generateClient(Site site) {
        HttpClientBuilder httpClientBuilder = HttpClients.custom();
        httpClientBuilder.setConnectionManager(this.connectionManager);
        if(site.getUserAgent() != null) {
            httpClientBuilder.setUserAgent(site.getUserAgent());
        } else {
            httpClientBuilder.setUserAgent("");
        }
        if(site.isUseGzip()) {
            httpClientBuilder.addInterceptorFirst(new HttpRequestInterceptor() {
                public void process(HttpRequest request, HttpContext context) throws HttpException, IOException {
                    if(!request.containsHeader("Accept-Encoding")) {
                        request.addHeader("Accept-Encoding", "gzip");
                    }
                }
            });
        }
        httpClientBuilder.setRedirectStrategy(new CustomRedirectStrategy());
        Builder socketConfigBuilder = SocketConfig.custom();
        socketConfigBuilder.setSoKeepAlive(true).setTcpNoDelay(true);
        socketConfigBuilder.setSoTimeout(site.getTimeOut());
        SocketConfig socketConfig = socketConfigBuilder.build();
        httpClientBuilder.setDefaultSocketConfig(socketConfig);
        this.connectionManager.setDefaultSocketConfig(socketConfig);
        httpClientBuilder.setRetryHandler(new DefaultHttpRequestRetryHandler(site.getRetryTimes(), true));
        this.generateCookie(httpClientBuilder, site);
        return httpClientBuilder.build();
    }

解析：上述代码主要是设置了UserAgent，Socket的一些基本配置，DefaultHttpRequestRetryHandler，HttpRequestInterceptor，ConnectionManager。
其中对于ConnectionManager的设置，我这边进行了深入的研究。

ConnectionManager

private PoolingHttpClientConnectionManager connectionManager;
   public HttpClientGenerator() {
       Registry<ConnectionSocketFactory> reg = RegistryBuilder.create().register("http", PlainConnectionSocketFactory.INSTANCE).register("https", this.buildSSLConnectionSocketFactory()).build();
       this.connectionManager = new PoolingHttpClientConnectionManager(reg);
       this.connectionManager.setDefaultMaxPerRoute(100);
       this.connectionManager.setMaxTotal(200);
   }

DefaultMaxPerRoute代表一个TCP连接最大可以连接的不同远程主机（ip+端口）最大数量为100

MaxTotal代表主机最大的连接数相当于最大可以开200个端口进行tcp连接。

PoolingHttpClientConnectionManager与BasicHttpClientConnectionManager的区别：
前者代表多线程，后者代表单线程。

HttpUriRequest

private HttpUriRequest convertHttpUriRequest(Request request, Site site, Proxy proxy) {
        RequestBuilder requestBuilder = this.selectRequestMethod(request).setUri(request.getUrl());
        if(site.getHeaders() != null) {
            Iterator var5 = site.getHeaders().entrySet().iterator();
            while(var5.hasNext()) {
                Entry<String, String> headerEntry = (Entry)var5.next();
                requestBuilder.addHeader((String)headerEntry.getKey(), (String)headerEntry.getValue());
            }
        }
        Builder requestConfigBuilder = RequestConfig.custom();
        if(site != null) {
            requestConfigBuilder.setConnectionRequestTimeout(site.getTimeOut()).setSocketTimeout(site.getTimeOut()).setConnectTimeout(site.getTimeOut()).setCookieSpec("standard");
        }
        if(proxy != null) {
            requestConfigBuilder.setProxy(new HttpHost(proxy.getHost(), proxy.getPort()));
        }
        requestBuilder.setConfig(requestConfigBuilder.build());
        HttpUriRequest httpUriRequest = requestBuilder.build();
        if(request.getHeaders() != null && !request.getHeaders().isEmpty()) {
            Iterator var7 = request.getHeaders().entrySet().iterator();
            while(var7.hasNext()) {
                Entry<String, String> header = (Entry)var7.next();
                httpUriRequest.addHeader((String)header.getKey(), (String)header.getValue());
            }
        }
        return httpUriRequest;
    }

主要是设置这次请求的Proxy和Header。

HttpClientContext

主要用于设置这次请求需要的cookie。

如果需要使用HttpClient池，并且想要做到一次登录的会话供多个HttpClient连接使用，就需要自己保存会话信息。因为客户端的会话信息是保存在cookie中的（JSESSIONID），所以只需要将登录成功返回的cookie复制到各个HttpClient使用即可。

使用Cookie的方法有两种，可以自己使用CookieStore来保存（见TestCookieStore()方法），也可以通过HttpClientContext上下文来维持（见TestContext()方法）。

使用CookieStore：

@Test
 public void testCookieStore() throws Exception {
   System.out.println("----testCookieStore");
   // 使用cookieStore方式
   CloseableHttpClient client = HttpClients.custom()
       .setDefaultCookieStore(cookieStore).build();
   HttpGet httpGet = new HttpGet(testUrl);
   System.out.println("request line:" + httpGet.getRequestLine());
   try {
     // 执行get请求
     HttpResponse httpResponse = client.execute(httpGet);
     System.out.println("cookie store:" + cookieStore.getCookies());
     printResponse(httpResponse);
   } catch (IOException e) {
     e.printStackTrace();
   } finally {
     try {
       // 关闭流并释放资源
       client.close();
     } catch (IOException e) {
       e.printStackTrace();
     }
   }
 }
 public static void printResponse(HttpResponse httpResponse)
     throws ParseException, IOException {
   // 获取响应消息实体
   HttpEntity entity = httpResponse.getEntity();
   // 响应状态
   System.out.println("status:" + httpResponse.getStatusLine());
   System.out.println("headers:");
   HeaderIterator iterator = httpResponse.headerIterator();
   while (iterator.hasNext()) {
     System.out.println("\t" + iterator.next());
   }
   // 判断响应实体是否为空
   if (entity != null) {
     String responseString = EntityUtils.toString(entity);
     System.out.println("response length:" + responseString.length());
     System.out.println("response content:"
         + responseString.replace("\r\n", ""));
   }
 }

使用context方式

@Test
  public void testContext() throws Exception {
    System.out.println("----testContext");
    // 使用context方式
    CloseableHttpClient client = HttpClients.createDefault();
    HttpGet httpGet = new HttpGet(testUrl);
    System.out.println("request line:" + httpGet.getRequestLine());
    try {
      // 执行get请求
      HttpResponse httpResponse = client.execute(httpGet, context);
      System.out.println("context cookies:"
          + context.getCookieStore().getCookies());
      printResponse(httpResponse);
    } catch (IOException e) {
      e.printStackTrace();
    } finally {
      try {
        // 关闭流并释放资源
        client.close();
      } catch (IOException e) {
        e.printStackTrace();
      }
    }
  }